State-tracked testing across network boundaries

ABSTRACT

Remote computing resource service providers, including online retailer, provide externally facing computer systems that allow users to interact with the service provider. Furthermore, the service provider may maintain computer systems and service inside an isolated network not exposed to users. Occasionally, service providers may test these externally facing computer systems using one or more external hosts operating on a public network. A coordinator may archive aggregated testing resources located on the isolated network and deploy the testing resources to the one or more external hosts. A database may be used to track the deployment state of the one or more external hosts in order to allow the coordinator to manage the testing process.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application incorporates by reference for all purposes the fulldisclosure of co-pending U.S. patent application Ser. No. 14/224,966,filed concurrently herewith, entitled “ACCESSING RESOURCES ACROSS ANETWORK BOUNDARY.”

BACKGROUND

The use of distributed computing systems and remote computing services,such as remote program execution and remote data storage, has greatlyincreased in recent years. Organization can provide scalable goods andservice to users and/or customer of the organization. For example, anorganization can provide a website, such as an electronic commercewebsite, to customers using distributed computing systems and remotecomputing services. Furthermore, organizations can employ virtualprivate networks (VPNs) to enable communication and improve securitybetween the distributed computing systems and remote computing services.It may be beneficial for organization to occasionally conduct tests,such as load test, on all or a portion of the distributed computingsystems and/or the remote computing services. However, it may bedifficult to accurately test the distributed computing systems andremote computing service of the organization from inside the VPNboundary. To complicate matters, test ran from outside the VPN may beunable to access all of the distributed computing systems and/or theremote computing services due to those distributed computing systemsand/or the remote computing services that are behind the VPN boundary.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 shows an example of a persistent state-tracked test in accordancewith at least one embodiment;

FIG. 2 show an example set up a persistent state-tracked test inaccordance with at least one embodiment;

FIG. 3A shows an example message diagram for a method for deploying apersistent, state-tracked testing environment across a VPN boundary inaccordance with at least one embodiment;

FIG. 3B shows an example message diagram for a method for deploying apersistent, state-tracked testing environment across a VPN boundary inaccordance with at least one embodiment;

FIG. 4 shows an example of accessing resources from outside a VPNboundary in accordance with at least one embodiment;

FIG. 5 shows an example message diagram for a method for accessing aresource from outside a VPN boundary in accordance with at least oneembodiment;

FIG. 6 shows an illustrative example of a process for deploying apersistent, state-tracked testing environment across a VPN boundary inaccordance with at least one embodiment;

FIG. 7 shows an illustrative example of a process for executing apersistent, state-tracked testing environment across a VPN boundary inaccordance with at least one embodiment;

FIG. 8 shows an illustrative example of a process for accessing aresource from outside a VPN boundary in accordance with at least oneembodiment; and

FIG. 9 illustrates an environment in which various embodiments can beimplemented.

DETAILED DESCRIPTION

In the following description, various embodiments will be described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the embodiments.However, it will also be apparent to one skilled in the art that theembodiments may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe embodiment being described.

Techniques described and suggested herein relate to persistent,state-tracked testing deployment across VPN boundaries and accessingresources across VPN boundaries to reverse various side effects of thetesting. In some embodiments, the testing system uses a single hostinside the VPN (referred to as a coordinator) to initiate a workflowthat persists state information in a distributed data store and tracksthe deployment, validation and extraction of a payload to one or moreexternal host operations on a public network. The payload may includeaggregated testing assets such as customer account number, productidentification number, customer information, product information,request information, host information, port information, distributedcomputing systems information, remote computing services, applicationdata, testing application, information or any other informationcorresponding to a system under test. The payload may be archived andstored in a storage system located on a public network in order toenable the one or more external hosts access to the archived payload.

The state of the one or more external hosts may be tracked using adatabase or other data storage system. The database may also be locatedon a public network in order to enable the one or more external hostsaccess to the database and allow the one or more external hosts totransmit state information to the database in order to be persisted. Thetesting system allows publicly-available services to be loaded orfunctionally tested using resources developed and stored within the VPN.The testing system may increase the quality of the test by moreaccurately simulating external client behavior through the use of theone or more hosts external to the VPN. Furthermore, by tracking thestate of the one or more external hosts in the external database, theconsistency of all external hosts is maintained and the workflow can beresumed from any source. The database may be initially configured by thecoordinator such that the database is populated with host names of theone or more external hosts and other information suitable formaintaining state information for the one or more external hosts.

Deployment of the test may be managed by one or more coordinatorsexecuting on computer systems inside the VPN. The coordinator maygenerate the testing environment by first collecting the testingresources (also referred to as testing parameters) from various otherdistributed computing systems and/or remote computing services insidethe VPN. The resources may be aggregated into an archive configured toenable the one or more external hosts to execute the test. The archivemay be stored in a remote data store accessible to the one or moreexternal hosts. Storing the archive may include the coordinator storingthe archive in a remote data store or the coordinator causing thearchive to be stored in the remote data store. The coordinator may thencall a virtual computing resource service in order to generate the oneor more external hosts. In various embodiments, the external hosts arealready executing and the coordinator transmits testing instructions tothe external hosts without first causing instantiation of the externalhosts. The virtual computing resource service may instantiate the one ormore external hosts, determine the one or more external hostsoperational state and transmit an indication to the coordinator that theone or more external hosts are operational. The one or more externalhosts may then transmit information corresponding to the current stateof the one or more external hosts to the database and the database mayrecord identification information for the one or more hosts and thetransmitted state information.

For example, the virtual machine management service may receive acommand to instantiate an external host, the virtual machine managementservice may then cause the external host to be instantiated and causethe external host to transmit a message indicating that the externalhost is operational once the external host has been instantiated.Causing the external host to transmit a message indicating that theexternal host is operational may include transmitting a request to theexternal hosts, the response to the request indicating that the externalhost is operational. The message may include identification informationfor the external host and information corresponding to the externalhost's current state. For example, the message may include a host namefor the external host and a message indicating the external host isoperational. In another example, the external host may receive a commandto download the archived payload, the command may also include anHypertext Transfer Protocol (HTTP) request to be executed by theexternal host upon completion of the download. The HTTP request maydirect the external host to the database and enable the external host toindicate the external host's current state. Returning to the exampleabove, if the external host has downloaded the archived payloadsuccessfully, the external may, using the HTTP request, indicate to thedatabase that the archived payload has been successfully downloaded andthe external host is ready to proceed with the deployment process.

The various states an external host, in an embodiment, include: on-line,prepared to receive payload, has received payload; prepared to validatepayload; has validated payload; prepared to extract payload; extractedpayload; prepared to validate extracted payload; has validated extractedpayload; and is prepared to execute test. At one or more of the steps inthe deployment process, the one or more external hosts may receivecommands over a Secure Shell (SSH) connection from the coordinatorindicating what step of the deployment process to execute. Thecoordinator may then query the database to determine which externalhosts of the one or more hosts have successfully executed the commandbased at least in part on the current status information contained inthe database. For example, the coordinator may transmit a command to theone or more external hosts, the command may indicate to the one or moreexternal hosts to validate the received payload. The one or moreexternal hosts may the validate the payload using a checksum andtransmit an indication to the database whether the payload validatedsuccessfully based at least in part on the checksum.

The coordinator may wait an amount of time and transmit a query to thedatabase in order to determine the current state of the one or moreexternal hosts. In various embodiments, the coordinator may subscribe toa stream operated by the database indicating received state informationfrom the one or more external hosts. If the coordinator determines thatany of the external hosts have failed a particular step in thedeployments process, the coordinator can cause the particular failedexternal host to roll back and retry the failed step. Furthermore, thecoordinator may perform various other mitigation techniques for externalhosts that have failed a particular step in the deployment processincluding retrying the failed step on all the external hosts, retryingthe entire deployment process, terminating the failed external hosts,replacing the failed external hosts with one or more replacementexternal hosts, failing the entire deployment process or any othertechnique suitable for mitigating the failed external hosts.

Once the one or more externals hosts have completed the deploymentprocess, the coordinator may transmit a command to the one or moreexternal hosts to perform various tests. The one or more external hostsmay all obtain identical payloads from the storage system but mayexecute different tests on the system under test. The one or moreexternal hosts may receive different test commands from the coordinator,for example, a subset of the one or more external hosts may receive acommand to execute a load test on the system under test and a secondsubset of the one or more external hosts may receive a command toexecute a purchase test on the system under test. The one or moreexternal hosts may execute multiple tests at once or in sequence basedon a single command or multiple commands from the coordinator.

Furthermore, the one or more external hosts may transmit informationcorresponding to the test to the database. For example, the one or moreexternal hosts may transmit the result of the test to the database andany change in state that results from the execution of the test. Thechange in state may include information such as the purchase history ofa customer, browse history of a customer, customer library, customercredit information, customer recommendations, third party creditinformation, advertisement information, customer entitlements, customerstatus information, increase in the quantity of items sold, reduction ininventory, information preventing the customer from purchasing the itemagain, item popularity, services the customer subscribes to, charges toa third party, charges to advertiser, experience points granted to thecustomer, changes to the customer's location, a discount or discountcode added to the customer account, reduction in the amount of customergift card or electronic wallet, notifications sent to ship an item orany other information corresponding to the state associated withcustomer account information. For example, a purchase test executed byone or more of the external hosts on the public network may result inthe purchase of an item being associated with one or more customeraccounts causing a change in the state of the one or more customeraccounts. By tracking the state of the customer accounts and the systemunder test, the state information may be reverted to a pre-test state sothat further tests may be run and the system under test is not alteredor otherwise affected by the tests executed by the coordinator and theone or more external hosts.

Contemporaneously or near contemporaneously with the execution of aparticular test, the one or more external hosts may enqueue a messagewith a queue operated by a queue service. The message may containsufficient information to revert the state information to a previousstate. The queue service may be listening to the queue in order todetect a message being enqueued into the queue. The queue service maythen obtain the message from the queue and cause a restore service torestore a previous state. For example, a particular external host may,during the execution of a purchase test, purchase an item which maycause a fulfillment service connected to the system under test to addthe item to a customer account associated with the particular externalhost. During the purchase test, the particular external host may enqueuea message with the queue indicating the item purchased and the customeraccount associated with the particular external host. The queue service,listening to the queue, may obtain the message and call the restoreservice with the information indicating the item purchased and thecustomer account associated with the particular external host. Therestore service may then access one or more systems inside the VPNboundary, such as the fulfillment service, and restore the state of thecustomer account to a previous state where the customer account does notinclude the purchase item.

FIG. 1 illustrates an example environment 100 where one or more computersystems running within a service provider environment, as well as theassociated code running thereon, may execute a deployment of testingresources outside of the service provider environment in order to testthe one or more computer systems of the service provider. A serviceprovider 118 may include website operators, online retailers, socialnetwork providers, cable providers, online game providers or any entitycapable of receiving automated agent traffic. A test coordinator 102(also referred to simply as a coordinator) may be located within a VPNboundary 116 of a service provider 118. The VPN boundary 116 may isolatecomputer systems connected to a private network from computer systemsconnected to a public network. For example, traffic from computersystems outside the VPN boundary 116 may be regulated such that some orall of the traffic may be restricted, allowed, partially allowed or anycombination thereof. The VPN boundary 116 may be implemented as networkinterface, computer system, computer subsystem, process, application orother such computer system entity. The coordinator 102 may be acollection of computing resources collectively configured to executevarious tests on computer system of the service provider 118. Forexample, the coordinator may be an internal host executed by the serviceprovider and operated by an administrator or other entity of the serviceprovider. The coordinator 102 may be connected to one or more systems orservices of the service provider 118 over a private network 114 such asthe system under test 104. Furthermore, one or more external hosts 106may connect to the online retailer 180 through the system under test 104and may initiate connection with and/or interaction with one or moreapplications running on the system under test 104 of the online retailer118 in order to execute various tests. The system under test 104 may bea collection of computing resources collectively configure to provideran externally facing interface for customers and/or users allowingentities to interact with the service provider 118. For example, thesystem under test 104 may be one or more webservers operated by theservice provider 118 in order to provide an interactive website forcustomers of the service provider 118. The system under test 104 mayalso be operating to provide services for the real accounts while thetest is occurring. For example, the one or more external hosts 106 maybe generating load on the system under test 104 contemporaneously withone or more customers accessing resources of the system under test 104in order to receive a service.

The service provider 118 through the system under test 104 may providean electronic marketplace offering goods and services for purchaseand/or consumption by customers of the service provider 118. The commandor commands to connect to the online retailer in order to execute thevarious tests may originate from an outside computer system and/orserver, or may originate from an entity, user or process on a remotenetwork location, or may originate from an entity, user or processwithin the service provider such as the test coordinator 102, or mayoriginate the one or more external hosts 106, or may originate as aresult of an automatic process or may originate as a result of acombination of these and/or other such origin entities. Informationcorresponding to the various tests may be included in an archivedpayload 110. The payload 110 may be generated by the coordinator 102 andmay include information suitable for executing the various tests by theone or more external hosts 106. In various embodiments, the payload 110may include one or more executable applications that, when executed bythe one or more external hosts 106, can cause the one or more externalhosts to perform the various tests. Various different archivingtechniques may be used to collect testing assets and generate thepayload 110 including concatenating testing assets, compressing testingassets, generating a software package containing the testing assets,generating a disk image containing the testing assets, encrypting thetesting assets or any combination thereof. For example, archiving thetesting assets to generate the payload may include concatenatingcustomer account information with a test application and compressing theresult of the concatenation into a single archive file.

In order to generate the payload, the coordinator 102 may aggregate datalocated within the VPN boundary 116, the data may include customeraccount information, product information and information correspondingto the system under test 104. For example, the data may include a numberof test customer accounts containing all the required accountinformation but the customer account may not correspond to an actualcustomer of the service provider 118. Furthermore, the data may includeall or a portion of the information corresponding to the goods and/orservices offered by the service provider 118. For example, the data mayinclude product identification numbers for all of the items offered forconsumption on an electronic commerce website operated by the serviceprovider 118. In another example, the data may include a list ofservices offered by the service provider. Finally, the data may alsoinclude information corresponding to the system under test 104 so as toenable the one or more external hosts to connect and communicate withthe system under test. For example, the data may include logininformation for the customer accounts, port information for the systemunder test 104, request format for the system under test 104 or anyother information suitable for execution of the test by the one or moreexternal hosts 106.

Once the payload data has been aggregated, the coordinator 102 maytransmit the payload to a storage system 122 located on the publicnetwork 112 or otherwise configured to be accessible outside the VPNboundary 116. The public network 112 may include, for example, a localnetwork, an internal network, a public network such as the Internet, awide-area network, a wireless network, a mobile network, a satellitenetwork, a distributed computing system with a plurality of networknodes and/or the like. The network may also operate in accordance withvarious protocols, such as those listed below, Bluetooth, WiFi, cellularnetwork protocols, satellite network protocols and/or others. In variousembodiments, the coordinator 102 transmits the payload 110 directly tothe one or more external hosts 106. The coordinator 102 may communicatewith the one or more external hosts 106 over a SSH connection or othersecured connection.

Once the payload 110 has been stored in the system 122, the coordinator102 may transmit a command to the one or more external hosts to obtainthe payload 110 from the storage system 122. The command may includeinformation capable of identifying the payload 110 and the storagesystem 122. For example, the command may include a uniform resourcelocator (URL) for the storage system 122 and the payload 110. Oncereceived, the payload 110 has been received by the one or more externalhosts 106, the one or more external hosts 106 may extract the payload110 and prepare to execute the various tests. The storage system 122 maybe a collection of computing resources configured to synchronouslyprocess requests to store and/or access data. The storage system 122 mayoperate using computing resources that enable the storage system 122 tolocate and retrieve data quickly, so as to allow data to be provided inresponses to requests for the data. For example, the storage system 122may maintain stored data in a manner such that, when a request for adata object is retrieved, the data object can be provided (or streamingof the data object can be initiated) in a response to the request. Asnoted, data stored in the storage system 122 may be organized into dataobjects. The data objects may have arbitrary sizes except, perhaps, forcertain constraints on size. Thus, the storage system 122 may storenumerous data objects of varying sizes. The storage system 122 mayoperate as a key value store that associates data objects withidentifiers of the data objects which may be used by the one or moreexternal hosts 106 to retrieve or perform other operations in connectionwith the data objects, such as the payload 110, stored by storage system122.

The one or more external hosts 106 may include one or more virtualmachines operated on a public network 112 by the service provider 118 ora service of the service provider 118, such as a virtual machineservice. For example, the coordinator may submit a request to thevirtual machine service in order to instantiate the one or more externalhosts 106 used to execute the various tests. In various embodiments, theone or more external hosts 106 are computer systems operated by thecustomers of the service provider 118. For example, the service provider118 may offer a discount or coupon for customers in exchanging forallowing the coordinator 102 to use the customers' computer systems asan external host 106 for executing the various tests. Once the one ormore external hosts 106 have been instantiated the virtual machineservice, the virtual machine service may transmit the host names to thecoordinator 102 and the coordinator 102 may populate the database 120with the host names in order to configure the database to receive statusdata 108 from the one or more external hosts 106. Furthermore, the oneor more external hosts 106 may be pre-instantiate computer systems,customer computer system where the customer has agreed in a terms ofservice agreement to allow the service provider to utilize thecustomer's computer system for testing purposes, mobile devices,tablets, gaming systems or any other device capable of generating loadon an externally facing computer system.

The virtual machine service may be a collection of computing resourcesconfigured to instantiate virtual machine instances on behalf of thecoordinator 102. The coordinator 102 may interact with the virtualmachine service (via appropriately configured and authenticated APIcalls) to provision and operate virtual computer systems that areinstantiated on physical computing devices hosted and operated by theservice provider 118 and located on a public network 112. The virtualmachines (also referred to a virtual computer systems) may be used forvarious purposes, such as to operate as servers supporting a website, tooperate business applications or, generally, to serve as computing powerfor the customer. Other applications for the virtual computer systemsmay be to support database applications, electronic commerceapplications, business applications and/or other applications. Althoughthe virtual machine service is used by the coordinator 102 in FIG. 1,any other computer system or computer system service may be utilized bythe coordinator 102, such as the coordinator that does not employvirtualization or instantiation and instead provisions computingresources on dedicated or shared computers/servers and/or other physicaldevices.

The one or more external hosts 106 may periodically or aperiodicallytransmit status data 108 (also referred to as state information) to thedatabase 120. For example, at each step of the deployment process theone or more external hosts 106 may transmit status data 110 to thedatabase 120. The database 120 may be a collection of computingresources that collectively operate to run one or more databases for thecoordinator. For example, the database 120 may maintain informationcorresponding to the one or more external hosts 106 for the coordinator102 such that the coordinator 120 may query the database 120 in order toobtain information corresponding to the one or more external hosts 106.The coordinator 102 may operate and manage the database 120 from thedatabase service by utilizing appropriately configured API calls.Furthermore, the one or more external hosts 106 may transmitinformation, such as the status data 108, to the database over thepublic network 112.

The status data 108 may include corresponding to the particular externalhost's 106 state during the deployment process. For example, aparticular external host 106 may receive a command to retrieve thepayload 110 from the storage system 122, based at least in part on thecommand, the particular external host may begin downloading the payload110 from the storage system 122. Upon completion of the download, theparticular external hosts 106 may transmit status data 108 to thedatabase 120, the status data indicating the particular external host106 has complete the download successfully. In the alternative, if theparticular external host 106 has failed to download the payload 110successfully, the status data may indicate that the download has failedand may also include error information corresponding to any errors thatmay have occurred during the download to cause the download to fail.

The coordinator 102 may query the database 120 in order to receive stateinformation corresponding to the one or more external hosts 106. Invarious embodiments, the database 120 transmits notifications to thecoordinator 102 indicating that status data 108 have been received fromthe one or more external hosts 106. The coordinator 102 may transmitcommands to the one or more external hosts 106 during the deployment andtesting process and may, at some point after transmitting the command,query the database 120 in order to determine the execution of thecommands by the one or more external hosts 106. For example, thecoordinator may transmit a command to the one or more external hosts 106requiring the one or more external hosts to validate the payload 110downloaded from the storage system 122. The command may include acandidate checksum configured to allow the one or more external hosts106 to validate the payload 110. The one or more external hosts 106 maythen validate the payload 110 and transmit information corresponding tovalidation of the payload 110 to the database 120. The coordinator maythen query the database 120 in order to determine the one or moreexternal hosts 106 that have executed the command to validate thepayload 110.

Based at least in part on the query results, the coordinator 102 mayperform a variety of different actions. For example, if all or asufficient percentage of the one or more external hosts 106 successfullyvalidated the payload 110, the coordinator 102 may proceed with the nextstep of the deployment process. Furthermore, if any of the one or moreexternal hosts 106 failed to validate the payload, the coordinator 102may cause the particular external host 106 that failed to validate thepayload to roll back to a previous state and attempt the command again.For example, the coordinator 102 may cause the particular external host106 to re-obtain the payload 110 again from the storage system 122 andrevalidate the payload based at least in part on the candidate checksum.Other actions may be performed by the coordinator 102 such asterminating external hosts 106 that have transmitted an indication tothe database 120 that a particular command has failed.

FIG. 2 illustrates an example environment 200 where one or more computersystems running within a service provider environment, as well as theassociated code running thereon, may execute a deployment of testingresources outside of the service provider environment in order to testthe one or more computer systems of the service provider. A serviceprovider 218 may execute a coordinator 202, the coordinator 202 maymanage the deployment and execution of various testing resources acrossa VPN boundary 216. The testing resources may include non-publicinformation maintained by the service provider 218, such as dummycustomer accounts used for testing various systems of the serviceprovider. The testing resources may further include a variety of otherinformation such as the test data described above in connection withFIG. 1. For example, the testing resources may include goods or servicesthe service provider 218 is attempting to test, such as mobileapplications offered for consumption on an application store or otherelectronic commerce site.

The coordinator 202 may collect testing resources, archive the testingresources and save the archived testing resources in the storage system222. The coordinator may collect the testing resources for a variety ofsources, including an account service to be discussed in greater detailbelow in connection with FIG. 5. In various embodiments, the coordinator202 is configured with the testing resources and is responsible formanaging the testing resources. The coordinator 202 may transmit thearchived testing resources to the storage system 222 through an API callcontaining the archived testing resources. Alternatively, thecoordinator 202 may also establish a secure connection with the storagesystem 222 in order to transmit the archived testing resources over thesecure connection. The coordinator 202 may also be configured tocommunicate with the one or more external hosts 206 over a secureconnection.

The one or more external hosts 206 may receive commands from the testcoordinator 202 and may transmit state information to the database 220.In various embodiments, the one or more external hosts 206 receive acommand from the test coordinator 202 through a virtual machine serviceas described above in connection with FIG. 1. For example, thecoordinator may submit service calls to the virtual machine service andthe virtual machine service may cause the one or more external hosts 206to execute the service calls submitted by the test coordinator 202. Theone or more external hosts 206 may transmit state information and otherinformation corresponding to commands received from the coordinator 202to the database 220. The one or more external hosts 206 may transmit thestate information and other information over a network connectionbetween the one or more external hosts and the database 220. Forexample, the commands received by the one or more external hosts 206from the coordinator 202 may include an HTTP request configured to allowthe one or more external hosts 206 to transmit the state information andother information to the database 220 over a network connection.

The storage system 222 and database 220 may include several separatedata tables, databases, data documents, dynamic data storage schemesand/or other data storage mechanisms and media for storing data relatingto a particular aspect of the present disclosure, such as the archivedtesting resources and the state information and other informationcorresponding to the one or more external hosts 206. The coordinator 202may obtain information from the database 220 in order to determine thestate and other information corresponding to the one or more externalhosts 206. The coordinator 202 may submit queries to the database 220using a variety of different techniques, for example, the coordinatormay submit a service call or API call to the database including a querycorresponding to the state and other information requested by thecoordinator. The coordinator 202 may also submit queries over a networkconnection with the database 220. In various embodiments, thecoordinator 202 submits queries and other requests, such as writerequests and database initialization requests, to a database managementservice responsible for managing the database 220.

FIGS. 3A-3B shows an example of a messaging diagram for deployingtesting resources across a VPN boundary using persistent state tracking300A. In FIGS. 3A-3B, a number of different components executing thedeployment are shown including a coordinator 302, a storage system 322,a host management service 304, external hosts 306 and a database 320.The coordinator 302 may be an internal host system as described above inconnection with FIG. 1. The storage system 322 may include a storagesystem capable of storing payload data as described above in connectionwith FIG. 1 The host management service 304 may be responsible forcoordinating the execution and operation of the external host 306. Forexample, the host management service 304 may be a virtual machineservice as described above in connection with FIG. 1. The external hostmay be one or more computer systems outside of the VPB boundary andconnected to a public network, such as the one or more external hostsdescribed above in connection with FIG. 1. The database 320 may be adata store configured to store information received from the one or moreexternal hosts, such as the database described above in connection withFIG. 2.

As shown in FIGS. 3A-3B, the coordinator 302 may deploy testingresources to external hosts in order to execute one or more tests oncomputer systems operated by the service provider. The coordinator 302may first archive the testing resources to enable the testing resourcesto be stored in storage system 322 and to be extracted and executed bythe external hosts 306. The coordinator 302 may execute a put command orservice call to put the payload 308 on the storage system 322. In oneexample, the coordinator 302 makes an API call including the payload tothe storage system 322, the API indicating to the storage system 322 tosave the payload. The storage system 322 may respond to the API call bycausing the payload to be stored and returning a location of the storedpayload. In various embodiments, the API call from the coordinatorincludes a location for the storage system 322 to store the payload. Thecoordinator may then transmit an update corresponding to the externalhost information 310 to the host management service 304. The externalhost information may include host names, host types, host operatingsystems, the total number of hosts or any other information suitable forinstantiating the external hosts 306.

The host management service 304 may then create the external hosts 312based at least in part on the external host information. The hostmanagement service 304 may transmit a command to one or more hypervisorsexecuted by the computer system of the service provider to instantiate aparticular number of external hosts 306. In various embodiments, thehost management service may monitor the instantiation of the externalhosts 306 in order to mitigate for any failures of the external hosts tosuccessfully instantiate. For example, the host management 304 servicemay monitor execution of the external host 306 and scale the computingcapacity of the external host upward or downward based at least in parton the execution of the external hosts. The external host 306 maytransmit an indication that the instantiation process is done 314 to thehost management service 304. If less than all the external hosts 306transmit the indication to the host management service 304, the hostmanagement service may then attempt to instantiate additional externalhosts 304 in order to satisfy the request by the coordinator 302. Invarious embodiments, the host management service does not attempt toinstantiate additional external hosts 306 to replace any external hosts306 that may have failed to successfully instantiate.

Once the external hosts 306 have been instantiated and the hostmanagement service 304 received the indication that the external hosts306 are operations, the host management service 304 may transmit anindication to the coordinator 302 that instantiation of the externalhosts 306 is done 316. The indication may include the number of externalhosts 306 that have successfully instantiated and the host namesassociated with each successfully instantiated external host 306. Ifless than the requested number of external hosts 306 were instantiated,the coordinator 302 may terminate the deployment process, requestadditional external hosts 306 or continue the deployment process withless than the requested number of external hosts 306. The coordinator302 may then populate the database 320 with external host information318. The external host information 318 may include identificationinformation such as host names for each of the external hosts 306. Thedatabase 320 or a database management service may then populate thedatabase 320 based at least in part on the external hosts informationand configure the database 320 to receive status and/or stateinformation from the external hosts 306.

The coordinator 302 may then transmit a command to the external hosts306 indicating to the external hosts 306 to get the payload data 324from the storage system 322. The command may include the location of thestorage system 322 and/or the location of the payload. Furthermore, thecommand may include an HTTP request to be executed by the external hosts306 upon completion of the command, the HTTP request may cause theexternal hosts 306 to update the database 320 with state information.The external hosts 306 may then request the payload from the storagesystem 322 by transmitting a get payload 326 request to the storagesystem 322. The storage system 322 may respond to the request bytransmitting the payload location 328 to the external host 306. Thepayload location may be a URL or other reference information configuredto refer to the location of the payload. Based at least in part on thelocation information associated with the payload, the external hosts 306may request download of the payload 330 from the storage system 322. Forexample, the external hosts may submit the URL along with a downloadrequest to the storage system 322.

Once the download is complete, the external hosts 306 may transmit theHTTP request included in the command from the coordinator 302 and updatethe database 320 with the state information corresponding to whether thedownload completed 332. At some point later, the coordinator 302 mayquery the database 320 to determine which external hosts 306 havesuccessfully downloaded the payload. In various embodiments, thedatabase 320 streams updates to the database 320 from the external hosts306 to the coordinator 302. The query from the coordinator 302 to thedatabase 320 may retrieve information corresponding to each externalhosts 306 and indicate whether the download completed 338. If theretrieved information indicates that any of the external hosts 306 havefailed, the coordinator 302 may retry 336 the download step on theexternal hosts 306 that failed to download the payload successfully.

Various other techniques may be used in accordance with the presentdisclosure to account for external hosts that have failed one or moresteps of the deployment process. For example, if the database 320indicates that a subset of the external hosts 306 have failed todownload the payload successfully, the coordinator 302 may retransmitthe download command to the subset of external hosts 306. Thecoordinator 302 may also cause any external hosts 306 that have failed aparticular step of the deployment process to roll-back to a previoussuccessful state, fail the entire deployment process, restart the entiredeployment process, cause a number of replacement external hosts 306 tobe instantiated in order to replace the failed external hosts 306 orproceed with the deployment process with attempting to retry orotherwise repair the failed external hosts 306. Furthermore, if thecoordinator 302 determines to proceed with the deployment withoutattempting to remediate any failed external hosts 306, the coordinator302 may still transmit the command to the failed external hosts 306. Forexample, if a particular external host 306 fails to validate theextracted payload, the coordinator 302 may still transmit the command tothe particular external hosts to perform various tests.

Returning to FIG. 3A, the coordinator 302 may determine, based at leastin part on the information retrieved from the database 320, to proceed340 with the deployment process. The deployment process is furtherillustrated in FIG. 3B. The coordinator 302 may transmit a command tothe external hosts 306 to validate a checksum 342, the checksumcorresponding to the payload obtained from the storage system 322. Thecommand may include a candidate checksum and an HTTP request to beexecuted after the external hosts 306 validate the candidate checksum,the HTTP request may be configured to update the database 320 with theresult of the validation. In various embodiments, the external hosts 306generate a checksum based at least inpart on the payload and transmitthe checksum to the coordinator for validation. Returning to FIG. 3B,the coordinator 203 may query the database 320 to determine if theexternal hosts 306 have validated the checksum 348. If the one or moreof the external hosts 306 have failed to valid the checksum, the failedexternal hosts 306 may retry 346 to validate the checksum or may retry346 the download operation and re-validate the checksum after thepayload has been downloaded from the storage system 322 again.

If all of the external hosts 306 have validated the checksum and/or ifthe coordinator has determined to proceed 350 with the deploymentprocess, the coordinator 302 may then transmit a command to the externalhosts 306 to extract the payload 352. The command may further include anHTTP request configured to be executed by the external hosts 306 andupdate the database 320. In various embodiments, the external hosts 306are configured to transmit state information to the database 320 withouta contemporaneous command from the coordinator 302. For example, theexternal hosts 306 may be configured to update the database 320 withstate information once the state of the external hosts 306 changes. Theexternal hosts 306 may then attempt to extract the payload, the payloadcontaining archived test assets as described above in connection withFIG. 1. For example, the payload may be a compressed archive containingthe testing application, information for formatting requests to thesystem under test, customer information, product information and anyother information suitable for testing externally facing computersystems of the service provider.

Once the external hosts 306 has attempted to extract the payload, theexternal hosts 306 may transmit the result of the extraction to thedatabase, the result may indicate whether the payload has been extracted354 successfully. The coordinator 302 may, at some point in time aftertransmitting the command to extract the payload 352, query the database320 to determine if the payload was extracted 358 successfully. If anyof the external hosts 306 have failed to extract the payloadsuccessfully, the coordinator may cause the external hosts 306 to retry356 extracting the payload. The coordinator 302 may cause the failedexternal hosts 306 to retry extracting the payload by re-transmittingthe command to extract the payload to the failed external hosts 306. Invarious embodiments, the failed external hosts 306 may retry anyparticular failed step of the deployment process without a retry commandfrom the coordinator.

If all of the external hosts 306 have extracted the payload successfullyand/or if the coordinator has determined to proceed 360 with thedeployment process, the coordinator 302 may then transmit a command tothe external hosts 306 to generate load 362. The external hosts 306 maygenerate load by performing a variety of tests. For example, aparticular test may have a corresponding test script, the test scriptwhen executed by the external hosts may cause the external hosts toperform a variety of different interactions with the system under test.The script may include logging on to the system under test, navigatingto a particular product webpage operated by the system under test,placing the item in an electronic shopping cart, purchasing the product,transmitting data to the system under test or other similar actionsexposed to users of the service provider. The external hosts 306 may beconfigured to execute multiple tests and to execute multiple testssimultaneously. Furthermore, the external hosts 306 may all receive thesame payload and execute different tests based at least in part on thesame payload. For example, a subset of the external host 306 may executea load test configured to simulate maximum peak traffic and a secondsubset of the external hosts 306 may execute a purchase test configuredto simulate customer purchasing items offered for sale on the serviceprovider's website. In various embodiments, the deployment process mayexecute in parallel on the external hosts 306, such that the externalhosts 306 may be in different states. The external hosts 306 may executedifferent deployment operation in parallel. For example, a firstexternal host may be executing a download command while a secondexternal host may be executing an extract payload command. Once asufficient number of external hosts have completed deployment, thevarious tests of the system under test may be executed.

FIG. 4 illustrates an example environment 400 where one or more computersystems running within a service provider environment, as well as theassociated code running thereon, may restore state information that mayhave been altered during the execution of one or more tests on thecomputer systems of the service provider. A service provider 418 mayoperate a queue 402, the queue 402 may be configured to receive messagesfrom one or more external hosts 406. The message may indicate operationsperformed by the external hosts 406 during the execution of one or moreload tests on the system under test 404. The load tests may be executedas a result of a command from a coordinator as described above inconnection with FIG. 1. Various deployment and testing processes may beused in accordance with the present disclosure along with or instead ofthe deployment process described above in connection with FIGS. 3A-3B.

During execution of various tests, the external hosts 406 may perform avariety of actions that alter or otherwise change state informationmaintained by the service provider. For example, the external hosts mayexecute a variety of purchase tests where the external hosts purchasesan item offered for consumption on a website operated by the serviceprovider. In response to the purchase, one or more systems of theservice provider operating within the VPN boundary 416 may performvarious actions, such as changing inventory information associated withthe purchase, charging a customer account associated with the purchase,debiting a gift card or electronic wallet associated with the purchase,adding items to a library associated with the purchase, preparing toship an item associated with the purchase and any other actionassociated with a purchase. The message enqueued by the external hosts406 in the queue 402 may indicate which state information and/or actionmay have been performed by computer systems of the service provider inresponse to the test.

The external host 406 may transmit these messages contemporaneously ornear contemporaneously with the execution of the test. For example, theexternal hosts 406 may execute a test and, as a result of the test,recommendation data may be generated based at least in part on theexecution of the test. After completion of the test, the external hosts406 may enqueue a message in the queue 402 indicating the test that theparticular test executed and the resulting state change (e.g., thegenerated recommendation data). In another example, the external hosts406 may, during execution of one or more tests, purchase a plurality ofmobile application through the system under test 404. A fulfillmentservice 408 may receive purchase requests and add the mobile applicationto a library of mobile applications associated with a customer account.The external hosts 406 may then enqueue one or more messages in thequeue 402 indicating the mobile applications purchased and theassociated customer accounts to enable the restore service 420 to removethe purchases from the associated customer accounts.

The queue 402 may be a collection of computing resources collectivelyconfigured to receive queue messages and allow other computer systems orservices to obtained enqueued messages. For example, the queue 402 maybe a message queue instantiated by a queue service 410, the queue 402further configured to receive messages over a public network and allowthe queue service 410 to listen for enqueued messages from a privatenetwork. The queue service 410 may be a collection of computingresources collectively configured to listen for enqueued messages andobtain the messages so that the messages can be processed by one or morecomputer systems of the service provider. For example, the queue service410 after instantiating the queue 402 may listen or otherwise receivedata from the queue 402 indicating that one or more messages from theexternal hosts 406 have been received and enable the restore service 420to process the message. Although the restore service 420 is illustratedin FIG. 4 as part of the queue service 410, the restore service 420 maybe separate from the queue service 410 in accordance with the presentdisclosure.

The restore service 420 may be a collection of computing resourcescollectively configured to receive queue messages and restore and/orcleanup altered state information indicated in the message. For example,the restore service 420 may be a computer system inside the VPN boundary416 configured to receive queue messages and transmit service calls tovarious other services of the service provider to restore the stateinformation to account for changes that may have occurred as a result ofthe tests executed by the external hosts 406. The restore service 420may receive a queue message from the queue service 410, the queuemessage may include information corresponding to a state changeresulting from the execution of a test and the restore service 420 maytransmit the information corresponding to the state change to an accountservice 412. The account service 412 may then cause the state to berestored based at least in part on the information corresponding to thestate change. For example, the enqueue message may contain a purchaseidentification number which is obtained by the restore service 420 andincluded in a service call to the account service 412. The accountservice 412 may cause state information corresponding to the purchaseidentification number to be deleted or otherwise restore the stateinformation to a previous state prior to the purchase being made.

The account service 412 may be a collection of computing resourcescollectively configured to receive information corresponding to acustomer account and perform a variety of actions including adding,deleting or restoring information. For example, the account service 412may be a computer system inside the VPN boundary responsible formanaging customer account information. The restore service 420 mayinteract with multiple computer systems or services of the serviceprovider in order to restore any state information that may have beenchanged during the execution of the tests by the external hosts 406. Forexample, the service provider may operate an inventory serviceresponsible for managing inventory information associated with itemsoffered for sale by the service provider. As a result of a particulartest executed by the external hosts 406, the inventory service maychange the inventory information associated with the items offered forsale. The restore service 420 may then transmit, based at least in parton a queue message indicating the inventory that has been altered by thetest, a service call to the inventory service to restore the inventoryinformation indicated in the queue message.

FIG. 5 shows an example of a messaging diagram for accessing privateresources from outside a VPN boundary. In FIG. 5 a number of differentcomponents executing the process are shown including external hosts 506,a restore service 520, a queue service 510, a system under test 508 andan account service 512. The external hosts 506 may include one or morecomputer systems as described above in connection with FIG. 1. Forexample, the external hosts may be configured to generate load on thesystem under test 508. The restore service 520 may be configured toreceive queue messages from the queue service 510 and cause stateinformation to be restored in multiple services of the service provideras described above in connection with FIG. 4. The queue service 510 maybe responsible for generating one or more queues configured to receivequeue messages from the external hosts 506 as described above inconnection with FIG. 4. The account service 512 may be responsible formaintaining state information corresponding to customers of the serviceprovider.

As show in FIG. 5, the restore service 520 may query the queue service510 in order to determine if one or more orders have been received 514.The example illustrated in FIG. 5 shows purchase orders as the stateinformation that may be altered as a result of the one or more tests,however any state information track by the service provider may bemonitored and restored in accordance with the present disclosure. Duringthe execution of the one or more tests on the system under test 508 itmay take some time before state information is altered by the externalhosts 506 executing the one or more tests. Furthermore, it may take timeonce the tests are executed for the external hosts 506 to enqueuemessages corresponding to the state information and for the queueservice 510 to obtain the messages from the queue. The queue service 510may then respond with an indication that no orders have been received518. The restore service may then wait 506 an amount of time beforetransmitting another query to determine if any purchase orders have beenreceived.

During the executing of a buy test or other similar test the externalhosts may generate a purchase order 522 through the system under test508. The system under test 508 may then call a fulfillment service 528in order to fulfill the purchase order submitted by the external hosts506. For example, the external hosts 506 may submit purchase orders fora movie to the system under test 508, the system under test 508 may thencall a fulfillment service 520 in order to add the movie to one or morecustomer libraries associated with purchase orders. The fulfillmentservice may return an order ID 526 associated with the purchase order.The order ID may include any identification information configured toidentify a particular purchase order submitted. The system under testmay then return the order ID to the external host 524. The system undertest 508 may return the order ID to the external host 506 as part of theconfirmation message indicating that the purchase order was processedsuccessfully. In various embodiments, the system under test 508 does notreturn the order ID to the external hosts 506 and enqueue the order IDdirectly with the queue associated with the queue service 510.

Once the external hosts 506 have received the order IDs associated withthe purchase order submitted, the external hosts 506 may then enqueuethe order IDs 530 with the queue in order to enable the restore serviceto receive the order IDs and cause the system under test 508 to havestate information restored based at least in part on the queue messages.The restore service 520 may then request queue messages indicating thatone or more purchase orders have been received 532 from the queueservice 510. The queue service 510 may obtain the queue message from thequeue prior to the request or may obtain the queue message as a resultof receiving the request. If no 536 queue messages are obtained by thequeue service 510, the restore service may again wait for an order to bereceived and transmit an additional request to the queue service 510 atsome point in time later. If messages have been enqueued by the externalhosts 506, the queue service 510 may transmit information correspondingto the messages to the restore service 520 or the queue service 510 maytransmit the messages directly to the restore service 520.

The restore service 520 may then log the order ID 540 and may thentransmit a revoke order ID 542 request to the account service 512. Theaccount service 512 may then revoke the purchase based at least in parton the order ID. Revocation of the purchase may restore the stateinformation maintained by the account service 512 or another service ofthe service provider to a previous state prior to the execution of thetests by the external hosts 506. For example, the purchase order may adda movie to a customer library associated with a customer account.Revocation of the purchase may remove the movie from the customerlibrary and allow the test to be executed again.

FIG. 6 shows an illustrative example of process 600 which may be used todeploy testing resources across a VPN boundary. The process 600 may beperformed by any suitable system such as the coordinator and virtualmachine service described above in connection with FIG. 2. Returning toFIG. 6, in an embodiment, the process 600 includes aggregating testassets 602. The coordinator may aggregate test assets required toexecute one or more load tests from a public network on externallyfacing computer systems of the service provider. The coordinator mayaggregate assets from a plurality of computer systems of the serviceprovider. The test assets may include application data, customer data,product data and any other data suitable for generating load on a systemunder test.

The coordinator may then archive the test assets 604. Archiving the testassets 604 may include generating a tar file of the test assets or othercompression technique suitable for aggregating the test assets into asingle file. Furthermore, the archived test assets may be stored in atemporary data store located on a public network such that the archivedtest assets are accessible from one or more external hosts. Thecoordinator may then cause one or more external hosts to be instantiated606. The coordinator may transmit a request to a virtual machine serviceto instantiate a number of external hosts on behalf of the coordinator.The virtual machine service may transmit an indication once the one ormore external hosts have been successfully instantiated. The coordinatormay then deploy the test assets to the one or more external hosts 608.Deploying the test assets to the one or more external hosts may includea variety of steps as shown in FIGS. 3A-3B. For example, the coordinatormay transmit various commands to the external hosts to download the testassets, validate the test assets and extract the test assets.

During deployment of the test assets the one or more external hosts maytransmit state information corresponding to the deployment process to adatabase associated with the coordinator. The coordinator may query thedatabase to determine external host failure 610. If any external hostshave failed 612 the deployment process the coordinator may perform avariety of mitigation actions. The coordinator may cause the failedexternal host to restart the deployment process until the deploymentprocess has completed successfully. The coordinator may also terminatethe failed external host and continue with the testing. In variousembodiments, the coordinator is configured to proceed with the testingwhen a quorum of external hosts have successfully completed thedeployment process. The coordinator may then transmit commands to theexternal hosts to execute the test 614 on the system under test.

FIG. 7 shows an illustrative example of the process 700 which may beused to deploy testing assets to one or more external hosts. The process700 may be performed by any suitable system such as the external hostsdescribed above in connection with FIG. 2. Returning to FIG. 7, in anembodiment, the process 700 includes receiving an archive of testingassets and sending a status update to a database 702. The external hostmay obtain an archive of testing assets (also referred to as a payload)from a storage system as described above in connection with FIG. 2.Furthermore, the external host may obtain the archive and send thestatus update based at least in part on a command from the coordinatorresponsible for managing the execution of the tests. The status updatemay indicate whether the archive was received successfully. Once thearchive is received, the external host may validate the archive and sendanother status update 704. The archive may be validated based at leastin part on a checksum received from the coordinator. The status updatemay be transmitted to a database associated with the coordinator and mayindicate whether the checksum received from the coordinator and thechecksum generated by the external host, based at least in part on thereceived archive, match.

The external hosts may then extract data from the validated archive andsend a status update 706. The external host may extract data from thearchive in order to execute a test application included in the archiveand/or use data contained in the archive to execute the test. Theextracted data may contain the test application that must be installedon the external hosts in order to perform the tests. Once the archiveddata is extracted the external host may be configured to execute thetest. All external hosts may receive that same archive or differentarchives and all external hosts may execute the same or different tests.Furthermore, the external host may execute multiple tests based on asingle archive file. The external host may then execute one or more testoperations and then send a status update 708 to the database. The statusupdate may include the results of the test and may also include anystate information that may have been changed as a result of the test.The service provider may then restore any state information that mayhave been changed as a result of the test based at least in part on thestatus update.

FIG. 8 shows an illustrative example of the process 800 which may be toaccess resources inside a VPN boundary from a public network in order toperform cleanup resulting from one or more test operations. The process800 may be performed by any suitable system such as the restore serviceand queue service described above in connection with FIG. 4. Returningto FIG. 8, in an embodiment, the process 800 includes receiving anoperation 802. The operation may be any operation that causes theservice provider to change or record information corresponding to thestate of the system under test. The operation may occur as a result of atest of the system under test, where the test is executed from a publicnetwork. For example, the operation may be a purchase, a subscription,an addition to a digital library, a debit operation or other similaroperation that causes the service provider to track informationcorresponding to the operation. The service provider may then change orrecord information based at least in part on the operation 804. Forexample, the fulfillment service, as described above in connection withFIG. 4, may receive a purchase order and change or record informationbased on the purchase order, such as inventory information correspondingto the purchase order. In another example, the fulfillment service mayadd information to a customer account indicating a subscription serviceto be added to the customer account.

The service provider may then transmit to the particular external hostresponsible for submitting the operation an indication that theoperation was performed 806. Returning to the example above, thefulfillment service may indicate in response to the purchase order thatthe purchase order was processed successfully. The particular externalhost may then enqueue the change information 808 or informationcorresponding to the operation. The external host may enqueue theoperation in a queue as described above in connection with FIG. 4. Oncethe operation has been enqueued a queue service may obtain the operationor information corresponding to the operation and forward the operationto a restore service. The restore service may then log the changeinformation 810, the change information may correspond to the changerecorded by the system under tests. For example, the change informationmay include information corresponding to an item purchase in response toa purchase test and the information recorded by the system under test inresponse to fulfilling the purchase. The restore service may maintain alog of the operations in order to track any information altered by theexecution of the tests and ensure that the information has been restoredor otherwise cleaned up. The log of the operations may also be used togenerate metrics data. The restore service or some other service of theservice provider may then restore or delete the information of thesystem under test based at least in part on the change information 812.For example, the restore service may transmit information correspondingto the operation to an account service, the account service may thendelete any state information recorded as a result of the one or moretests executed by the particular external host.

FIG. 9 illustrates aspects of an example environment 900 forimplementing aspects in accordance with various embodiments. As will beappreciated, although a web-based environment is used for purposes ofexplanation, different environments may be used, as appropriate, toimplement various embodiments. The environment includes an electronicclient device 902, which can include any appropriate device operable tosend and/or receive requests, messages or information over anappropriate network 904 and, in some embodiments, convey informationback to a user of the device. Examples of such client devices includepersonal computers, cell phones, handheld messaging devices, laptopcomputers, tablet computers, set-top boxes, personal data assistants,embedded computer systems, electronic book readers and the like. Thenetwork can include any appropriate network, including an intranet, theInternet, a cellular network, a local area network, a satellite networkor any other such network and/or combination thereof. Components usedfor such a system can depend at least in part upon the type of networkand/or environment selected. Protocols and components for communicatingvia such a network are well known and will not be discussed herein indetail. Communication over the network can be enabled by wired orwireless connections and combinations thereof. In this example, thenetwork includes the Internet, as the environment includes a web server906 for receiving requests and serving content in response thereto,although for other networks an alternative device serving a similarpurpose could be used as would be apparent to one of ordinary skill inthe art.

The illustrative environment includes at least one application server908 and a data store 910. It should be understood that there can beseveral application servers, layers or other elements, processes orcomponents, which may be chained or otherwise configured, which caninteract to perform tasks such as obtaining data from an appropriatedata store. Servers, as used herein, may be implemented in various ways,such as hardware devices or virtual computer systems. In some contexts,servers may refer to a programming module being executed on a computersystem. As used herein, unless otherwise stated or clear from context,the term “data store” refers to any device or combination of devicescapable of storing, accessing and retrieving data, which may include anycombination and number of data servers, databases, data storage devicesand data storage media, in any standard, distributed, virtual orclustered environment. The application server can include anyappropriate hardware, software and firmware for integrating with thedata store as needed to execute aspects of one or more applications forthe client device, handling some or all of the data access and businesslogic for an application. The application server may provide accesscontrol services in cooperation with the data store and is able togenerate content including, but not limited to, text, graphics, audio,video and/or other content usable to be provided to the user, which maybe served to the user by the web server in the form of HyperText MarkupLanguage (“HTML”), Extensible Markup Language (“XML”), JavaScript,Cascading Style Sheets (“CSS”) or another appropriate client-sidestructured language. Content transferred to a client device may beprocessed by the client device to provide the content in one or moreforms including, but not limited to, forms that are perceptible to theuser audibly, visually and/or through other senses including touch,taste, and/or smell. The handling of all requests and responses, as wellas the delivery of content between the client device 902 and theapplication server 908, can be handled by the web server using PHP:Hypertext Preprocessor (“PHP”), Python, Ruby, Perl, Java, HTML, XML oranother appropriate server-side structured language in this example. Itshould be understood that the web and application servers are notrequired and are merely example components, as structured code discussedherein can be executed on any appropriate device or host machine asdiscussed elsewhere herein. Further, operations described herein asbeing performed by a single device may, unless otherwise clear fromcontext, be performed collectively by multiple devices, which may form adistributed and/or virtual system.

The data store 910 can include several separate data tables, databases,data documents, dynamic data storage schemes and/or other data storagemechanisms and media for storing data relating to a particular aspect ofthe present disclosure. For example, the data store illustrated mayinclude mechanisms for storing production data 912 and user information916, which can be used to serve content for the production side. Thedata store also is shown to include a mechanism for storing log data914, which can be used for reporting, analysis or other such purposes.It should be understood that there can be many other aspects that mayneed to be stored in the data store, such as page image information andaccess rights information, which can be stored in any of the abovelisted mechanisms as appropriate or in additional mechanisms in the datastore 910. The data store 910 is operable, through logic associatedtherewith, to receive instructions from the application server 908 andobtain, update or otherwise process data in response thereto. Theapplication server 908 may provide static, dynamic or a combination ofstatic and dynamic data in response to the received instructions.Dynamic data, such as data used in web logs (blogs), shoppingapplications, news services and other such applications may be generatedby server-side structured languages as described herein or may beprovided by a content management system (“CMS”) operating on, or underthe control of, the application server. In one example, a user, througha device operated by the user, might submit a search request for acertain type of item. In this case, the data store might access the userinformation to verify the identity of the user and can access thecatalog detail information to obtain information about items of thattype. The information then can be returned to the user, such as in aresults listing on a web page that the user is able to view via abrowser on the user device 902. Information for a particular item ofinterest can be viewed in a dedicated page or window of the browser. Itshould be noted, however, that embodiments of the present disclosure arenot necessarily limited to the context of web pages, but may be moregenerally applicable to processing requests in general, where therequests are not necessarily requests for content.

Each server typically will include an operating system that providesexecutable program instructions for the general administration andoperation of that server and typically will include a computer-readablestorage medium (e.g., a hard disk, random access memory, read onlymemory, etc.) storing instructions that, when executed by a processor ofthe server, allow the server to perform its intended functions. Suitableimplementations for the operating system and general functionality ofthe servers are known or commercially available and are readilyimplemented by persons having ordinary skill in the art, particularly inlight of the disclosure herein.

The environment, in one embodiment, is a distributed and/or virtualcomputing environment utilizing several computer systems and componentsthat are interconnected via communication links, using one or morecomputer networks or direct connections. However, it will be appreciatedby those of ordinary skill in the art that such a system could operateequally well in a system having fewer or a greater number of componentsthan are illustrated in FIG. 9. Thus, the depiction of the system 900 inFIG. 9 should be taken as being illustrative in nature and not limitingto the scope of the disclosure.

The various embodiments further can be implemented in a wide variety ofoperating environments, which in some cases can include one or more usercomputers, computing devices or processing devices which can be used tooperate any of a number of applications. User or client devices caninclude any of a number of general purpose personal computers, such asdesktop, laptop or tablet computers running a standard operating system,as well as cellular, wireless and handheld devices running mobilesoftware and capable of supporting a number of networking and messagingprotocols. Such a system also can include a number of workstationsrunning any of a variety of commercially-available operating systems andother known applications for purposes such as development and databasemanagement. These devices also can include other electronic devices,such as dummy terminals, thin-clients, gaming systems and other devicescapable of communicating via a network. These devices also can includevirtual devices such as virtual machines, hypervisors and other virtualdevices capable of communicating via a network.

Various embodiments of the present disclosure utilize at least onenetwork that would be familiar to those skilled in the art forsupporting communications using any of a variety ofcommercially-available protocols, such as Transmission ControlProtocol/Internet Protocol (“TCP/IP”), User Datagram Protocol (“UDP”),protocols operating in various layers of the Open System Interconnection(“OSI”) model, File Transfer Protocol (“FTP”), Universal Plug and Play(“UpnP”), Network File System (“NFS”), Common Internet File System(“CIFS”) and AppleTalk. The network can be, for example, a local areanetwork, a wide-area network, a virtual private network, the Internet,an intranet, an extranet, a public switched telephone network, aninfrared network, a wireless network, a satellite network and anycombination thereof.

In embodiments utilizing a web server, the web server can run any of avariety of server or mid-tier applications, including Hypertext TransferProtocol (“HTTP”) servers, FTP servers, Common Gateway Interface (“CGI”)servers, data servers, Java servers, Apache servers and businessapplication servers. The server(s) also may be capable of executingprograms or scripts in response to requests from user devices, such asby executing one or more web applications that may be implemented as oneor more scripts or programs written in any programming language, such asJava®, C, C# or C++, or any scripting language, such as Ruby, PHP, Perl,Python or TCL, as well as combinations thereof. The server(s) may alsoinclude database servers, including without limitation thosecommercially available from Oracle®, Microsoft®, Sybase® and IBM® aswell as open-source servers such as MySQL, Postgres, SQLite, MongoDB,and any other server capable of storing, retrieving and accessingstructured or unstructured data. Database servers may includetable-based servers, document-based servers, unstructured servers,relational servers, non-relational servers or combinations of theseand/or other database servers.

The environment can include a variety of data stores and other memoryand storage media as discussed above. These can reside in a variety oflocations, such as on a storage medium local to (and/or resident in) oneor more of the computers or remote from any or all of the computersacross the network. In a particular set of embodiments, the informationmay reside in a storage-area network (“SAN”) familiar to those skilledin the art. Similarly, any necessary files for performing the functionsattributed to the computers, servers or other network devices may bestored locally and/or remotely, as appropriate. Where a system includescomputerized devices, each such device can include hardware elementsthat may be electrically coupled via a bus, the elements including, forexample, at least one central processing unit (“CPU” or “processor”), atleast one input device (e.g., a mouse, keyboard, controller, touchscreen or keypad) and at least one output device (e.g., a displaydevice, printer or speaker). Such a system may also include one or morestorage devices, such as disk drives, optical storage devices andsolid-state storage devices such as random access memory (“RAM”) orread-only memory (“ROM”), as well as removable media devices, memorycards, flash cards, etc.

Such devices also can include a computer-readable storage media reader,a communications device (e.g., a modem, a network card (wireless orwired), an infrared communication device, etc.) and working memory asdescribed above. The computer-readable storage media reader can beconnected with, or configured to receive, a computer-readable storagemedium, representing remote, local, fixed and/or removable storagedevices as well as storage media for temporarily and/or more permanentlycontaining, storing, transmitting and retrieving computer-readableinformation. The system and various devices also typically will includea number of software applications, modules, services or other elementslocated within at least one working memory device, including anoperating system and application programs, such as a client applicationor web browser. It should be appreciated that alternate embodiments mayhave numerous variations from that described above. For example,customized hardware might also be used and/or particular elements mightbe implemented in hardware, software (including portable software, suchas applets) or both. Further, connection to other computing devices suchas network input/output devices may be employed.

Storage media and computer readable media for containing code, orportions of code, can include any appropriate media known or used in theart, including storage media and communication media, such as, but notlimited to, volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information such as computer readable instructions, data structures,program modules or other data, including RAM, ROM, Electrically ErasableProgrammable Read-Only Memory (“EEPROM”), flash memory or other memorytechnology, Compact Disc Read-Only Memory (“CD-ROM”), digital versatiledisk (DVD) or other optical storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices or any othermedium which can be used to store the desired information and which canbe accessed by the system device. Based on the disclosure and teachingsprovided herein, a person of ordinary skill in the art will appreciateother ways and/or methods to implement the various embodiments.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims.

Other variations are within the spirit of the present disclosure. Thus,while the disclosed techniques are susceptible to various modificationsand alternative constructions, certain illustrated embodiments thereofare shown in the drawings and have been described above in detail. Itshould be understood, however, that there is no intention to limit theinvention to the specific form or forms disclosed, but on the contrary,the intention is to cover all modifications, alternative constructionsand equivalents falling within the spirit and scope of the invention, asdefined in the appended claims.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the disclosed embodiments (especially in thecontext of the following claims) are to be construed to cover both thesingular and the plural, unless otherwise indicated herein or clearlycontradicted by context. The terms “comprising,” “having,” “including”and “containing” are to be construed as open-ended terms (i.e., meaning“including, but not limited to,”) unless otherwise noted. The term“connected,” when unmodified and referring to physical connections, isto be construed as partly or wholly contained within, attached to orjoined together, even if there is something intervening. Recitation ofranges of values herein are merely intended to serve as a shorthandmethod of referring individually to each separate value falling withinthe range, unless otherwise indicated herein and each separate value isincorporated into the specification as if it were individually recitedherein. The use of the term “set” (e.g., “a set of items”) or “subset”unless otherwise noted or contradicted by context, is to be construed asa nonempty collection comprising one or more members. Further, unlessotherwise noted or contradicted by context, the term “subset” of acorresponding set does not necessarily denote a proper subset of thecorresponding set, but the subset and the corresponding set may beequal.

Conjunctive language, such as phrases of the form “at least one of A, B,and C,” or “at least one of A, B and C,” unless specifically statedotherwise or otherwise clearly contradicted by context, is otherwiseunderstood with the context as used in general to present that an item,term, etc., may be either A or B or C, or any nonempty subset of the setof A and B and C. For instance, in the illustrative example of a sethaving three members, the conjunctive phrases “at least one of A, B, andC” and “at least one of A, B and C” refer to any of the following sets:{A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of A, at least one of B and at least one of C eachto be present.

Operations of processes described herein can be performed in anysuitable order unless otherwise indicated herein or otherwise clearlycontradicted by context. Processes described herein (or variationsand/or combinations thereof) may be performed under the control of oneor more computer systems configured with executable instructions and maybe implemented as code (e.g., executable instructions, one or morecomputer programs or one or more applications) executing collectively onone or more processors, by hardware or combinations thereof. The codemay be stored on a computer-readable storage medium, for example, in theform of a computer program comprising a plurality of instructionsexecutable by one or more processors. The computer-readable storagemedium may be non-transitory.

The use of any and all examples, or exemplary language (e.g., “such as”)provided herein, is intended merely to better illuminate embodiments ofthe invention and does not pose a limitation on the scope of theinvention unless otherwise claimed. No language in the specificationshould be construed as indicating any non-claimed element as essentialto the practice of the invention.

Preferred embodiments of this disclosure are described herein, includingthe best mode known to the inventors for carrying out the invention.Variations of those preferred embodiments may become apparent to thoseof ordinary skill in the art upon reading the foregoing description. Theinventors expect skilled artisans to employ such variations asappropriate and the inventors intend for embodiments of the presentdisclosure to be practiced otherwise than as specifically describedherein. Accordingly, the scope of the present disclosure includes allmodifications and equivalents of the subject matter recited in theclaims appended hereto as permitted by applicable law. Moreover, anycombination of the above-described elements in all possible variationsthereof is encompassed by the scope of the present disclosure unlessotherwise indicated herein or otherwise clearly contradicted by context.

All references, including publications, patent applications and patents,cited herein are hereby incorporated by reference to the same extent asif each reference were individually and specifically indicated to beincorporated by reference and were set forth in its entirety herein.

What is claimed is:
 1. A computer-implemented method for deployingtesting resources, comprising: under the control of one or more computersystems configured with executable instructions, generating an archive,the archive including a testing application and testing parameters;storing the archive in a storage system such that the archive isaccessible from a public network; populating a database with informationcorresponding to one or more external hosts, the database configured tomaintain state information corresponding to the one or more externalhosts where the state information indicates whether a particularoperation of a deployment process has completed successfully;determining a set of external hosts to be used as hosts of the one ormore external hosts; deploying the archive to the set of external hostsover the public network, thereby causing the set of external hosts totransmit state information to the database during deployment; queryingthe database to determine state information corresponding to at least asubset of the set of external hosts, the state information indicatingthat the subset of the set of external hosts completed deploymentsuccessfully; and transmitting one or more commands to the set ofexternal hosts to test a system under test.
 2. The computer-implementedmethod of claim 1, wherein the computer-implemented method furtherincludes instantiating the set of external hosts such that the set ofexternal hosts are connected to the public network.
 3. Thecomputer-implemented method of claim 1, wherein testing parametersfurther include customer account information, product information, andrequest format information.
 4. The computer-implemented method of claim1, wherein: querying the database to determine state informationcorresponding further includes receiving, in response to the query, anindication that at least a subset of the set of external hosts failedduring deployment; and transmitting another command to the subset of theset of external hosts, the other command indicating to the external hostto retry deployment of the archive.
 5. A system, comprising: one or moreprocessors; and memory with instructions that, when executed by the oneor more processors, cause the system to: populate a database withinformation corresponding to one or more external hosts; determine a setof external hosts to be used as hosts of the one or more external hosts;transmit a first command to the set of external hosts to obtain anarchive from a storage system; transmit one or more other commands tothe set of external hosts to deploy the archive to the set of externalhosts such that the set of external hosts are configured to transmitstate information to the database during execution of the one or moreother commands; query the database to determine if deployment completedsuccessfully; and transmit a second command to the set of external hoststo execute a test based at least in part on the determination.
 6. Thesystem of claim 5, wherein the one or more commands include a request toupdate that database, the request when executed by the set of externalhosts cause the set of external hosts to transmit state information tothe database.
 7. The system of claim 5, wherein the memory furtherincludes instructions that when executed by the one or more processorcause the system to: determine, based at least in part on the query, asubset of the set of external hosts that failed a particular operationof deployment; and transmit a third command to the subset of the set ofexternal hosts to retry the particular operation of deployment thatfailed.
 8. The system of claim 5, wherein the determination thatdeployment completed successfully includes receiving an indication, inresponse to the query, that a certain percentage of the set of externalhosts completed deployment successfully.
 9. The system of claim 5,wherein the second command includes a request, the request when executedby the set of external hosts cause the set of external hosts to transmitinformation corresponding to execution of the test to the database. 10.The system of claim 5, wherein at least one of the one or more othercommands includes a candidate checksum useable to validate the archive.11. The system of claim 5, wherein the memory further includesinstructions that when executed by the one or more processors cause thesystem to: transmit a third command to a first subset of the set ofexternal hosts to execute a first test based at least in part on thedetermination; and transmit a fourth command to a second subset of theset of external hosts to execute a second test based at least in part onthe determination.
 12. The system of claim 5, wherein the memory furtherincludes instructions that when executed by the one or more processorcause the system to: query the database to determine if the set ofexternal hosts have successfully completed a particular operation ofdeployment; and if the set of external hosts have successfully completedthe particular operation, based at least in part on a response to thequery, transmit a next command of the one or more commands to the set ofexternal hosts, the next command indicating to the set of external hoststo proceed with deployment.
 13. A non-transitory computer-readablestorage medium having collectively stored thereon executableinstructions that, when executed by one or more processors of a computersystem, cause the computer system to at least: populate a database withhost information corresponding to a set of hosts; deploy archivedtesting parameters by at least: determining a set of computer systems tobe used as hosts of the set of hosts; transmitting one or more commandsto the set of computer systems, the one or more commands causing the setof computer systems to deploy the archived testing parameters; andreceiving, at the database, one or more updates corresponding tocompletion of the one or more commands; and transmit an execute commandto the set of computer systems based at least in part on a queryresponse from the database indicating that the one or more commandscompleted successfully, the execute command causing the set of hosts toexecute one or more tests based at least in part on the archived testingparameters.
 14. The non-transitory computer-readable storage medium ofclaim 13, wherein the instructions further comprise instructions that,when executed by the one or more processors, cause the computer systemto instantiate the set of computer systems based at least in part on afirst command from the computer system.
 15. The non-transitorycomputer-readable storage medium of claim 13, wherein the instructionsthat cause the computer system to determine the set of computer systemsto be used as hosts of the set of computer systems further includeinstructions that cause the computer system to make the determinationbased at least in part on the set of customer computer systems beingconnected to a public network.
 16. The non-transitory computer-readablestorage medium of claim 13, wherein the instructions further compriseinstructions that, when executed by the one or more processors, causethe computer system to: determine, based at least in part on the one ormore updates, at least one host of the set of computer systems thatfailed a particular command of the one or more commands; and transmit aretry command to the at least one host, the retry command causing the atleast one host to retry the particular command.
 17. The non-transitorycomputer-readable storage medium of claim 13, wherein the instructionsthat cause the computer system to archive testing parameters in astorage system further include instructions that cause the computersystem to retrieve the testing parameters from an isolated network. 18.The non-transitory computer-readable storage medium of claim 13, whereinthe instructions that cause the computer system to transmit one or morecommands to the set of computer systems further include instructionsthat cause the computer system to transmit one or more requestsassociated with the one or more commands, the one or more requestsindicating to the set of computer systems to transmit the one or moreupdates to the database.
 19. The non-transitory computer-readablestorage medium of claim 13, wherein the instructions further compriseinstructions that, when executed by the one or more processors, causethe computer system to: query the database to determine a subset of theset of computer systems that failed a particular command of the one ormore commands; and transmit, based at least in part on thedetermination, a retry command to the set of computer systems, the retrycommand indicating to the set of computer systems to retry the one ormore commands causing the set of computer systems to deploy the archivedtesting parameters.
 20. The non-transitory computer-readable storagemedium of claim 13, wherein the instructions that cause the computersystem to deploy archived testing parameters further includeinstructions that cause the computer system to deploy archived testingparameters further including customer account information, productinformation, and request format information.